Gaze-based coordination of virtual effects indicators

ABSTRACT

A method and system for providing gaze-based generation of virtual effects indicators correlated with directional sounds is disclosed. Gaze data is tracked via a camera associated with a client device to identify a point of focus within a three-dimensional virtual environment towards which one or both eyes of the player are focused. When the point of focus indicated by the gaze data when the point of focus does not move towards the source location within the three-dimensional virtual environment when the directional sound is received indicates that a virtual effect indicator associated with the directional sound type of the indicated directional sound is should be generated.

BACKGROUND OF THE INVENTION 1. Field of the Disclosure

The present technology pertains to detecting gaze to generate virtualeffects indicators correlated with directional sounds. Morespecifically, the present technology may provide for dynamic generationof the virtual effects indicator during gameplay based on the detectedgaze.

2. Description of the Related Art

Presently available interactive media titles, including virtual realitymedia titles, still face a wide range of accessibility issues pertainingto sensory disabilities such as blindness and low vision, hearing loss,deafness, deaf-blindness, and other sensory processing disorders.Advances in technology, driven, for example, by improving gameplayexperiences for disabled players have resulted in a more inclusiveenvironment, but there is still a lot of room for improvement.

In many interactive media titles, directional sound is used to conveyincoming characters or action and are key to the storyline. Missing suchdirectional sound due to sensory disabilities may result in aless-than-whole experience for some players.

Therefore, there is a need to provide a service that provides virtualeffect indicators based on certain cues that the player is not hearingvarious directional sounds using gaze-based detection.

SUMMARY OF THE CLAIMED INVENTION

Aspects of the present disclosure include systems and methods fordetecting gaze to generate virtual effects indicators correlated withdirectional sounds. Information regarding a plurality of directionalsound types associated with one or more interactive content titles maybe stored in memory. Each directional sound type may be associated withone or more virtual effects indicators in a respective interactivecontent title. Gameplay data sent over a communication network from aclient device of a player engaged in a current activity of one of theinteractive content titles within a current gameplay session may bemonitored. The gameplay data may be indicative of a directional soundassociated with a source location within a three-dimensional virtualenvironment of the interactive content title.

Gaze data may be tracked via a camera associated with the client deviceduring the current gameplay session to identify a point of focus withinthe three-dimensional virtual environment towards which one or both eyesof the player are focused. When the point of focus indicated by the gazedata does not move towards the source location within thethree-dimensional virtual environment when the gameplay data indicativeof the directional sound is received, a virtual effect indicatorassociated with the directional sound type of the indicated directionalsound may be generated to be presented within a display of the clientdevice. The virtual effect indicator indicates the source location ofthe indicated directional sound.

Virtual effect indicators may include at least one of a visual dopplereffect centered around the source location of the directional sound onthe display, a written cue indicating which direction to look toward thesource location, a visual indicator pointing to the source location ofthe directional sound or a direction of the directional sound, and anotification alerting the player to the source location of thedirectional sound. The gaze data may be mapped as the point of focusmoves to a plurality of locations in the three-dimensional virtualenvironment. The point of focus indicated by the gaze data may beidentified as not moving towards the source location within thethree-dimensional virtual environment based on the mapped gaze data.

Identifying that the point of focus indicated by the gaze data does notmove towards the source location within the three-dimensional virtualenvironment may further include comparing one or more three-dimensionalcoordinates associated with the source location of the directional soundwith one or more three-dimensional coordinates associated with thedetermined point of focus. Alternatively or in addition to, identifyingthat the point of focus indicated by the gaze data does not move towardsthe source location within the three-dimensional virtual environment mayinclude determining the orientation of the player within thethree-dimensional virtual environment and determining a player field ofvision that includes one or more three-dimensional coordinates ofthree-dimensional virtual environment based on the determinedorientation. Other three-dimensional coordinates associated with thethree-dimensional virtual environment may be excluded from the playerfield of vision.

Various aspects of the present disclosure may include methods ofgaze-based generation of virtual effects indicators correlated withdirectional sounds. Such methods may include storing informationregarding a plurality of directional sound types associated with one ormore interactive content titles in memory, each directional sound typeassociated with one or more virtual effects indicators in a respectiveinteractive content title. Such methods may include monitoring gameplaydata sent over a communication network from a client device of a playerengaged in a current activity of one of the interactive content titleswithin a current gameplay session, wherein the gameplay data isindicative of a directional sound associated with a source locationwithin a three-dimensional virtual environment of the interactivecontent titles.

Such methods may include tracking gaze data via a camera associated withthe client device during the current gameplay session to identify apoint of focus within the three-dimensional virtual environment towardswhich one or both eyes of the player are focused. Such methods mayinclude identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three-dimensionalvirtual environment when the gameplay data indicative of the directionalsound is received. Such methods may include generating a virtual effectindicator associated with the directional sound type of the indicateddirectional sound to present within a display of the client device,wherein the virtual effect indicator indicates the source location ofthe indicated directional sound.

Additional aspects of the present disclosure may include systems forgaze-based generation of virtual effects indicators correlated withdirectional sounds. Such systems may include memory that storesinformation regarding a plurality of directional sound types associatedwith one or more interactive content titles in memory, each directionalsound type associated with one or more virtual effects indicators in arespective interactive content title. Such system may include one ormore processors that executes instructions stored in memory. Executionof the instructions by the one or more processors may monitor gameplaydata sent over a communication network from a client device of a playerengaged in a current activity of one of the interactive content titleswithin a current gameplay session, wherein the gameplay data isindicative of a directional sound associated with a source locationwithin a three-dimensional virtual environment of the interactivecontent titles. Execution of the instructions by the one or moreprocessors may track gaze data via a camera associated with the clientdevice during the current gameplay session to identify a point of focuswithin the three-dimensional virtual environment towards which one orboth eyes of the player are focused.

Execution of the instructions by the one or more processors may identifythat the point of focus indicated by the gaze data does not move towardsthe source location within the three-dimensional virtual environmentwhen the gameplay data indicative of the directional sound is received.The one or more object-object associations may dynamically generate theplay data based on the one or more object-object associations to bedisplayed along with the media. Execution of the instructions by the oneor more processors may generate a virtual effect indicator associatedwith the directional sound type of the indicated directional sound topresent within a display of the client device, wherein the virtualeffect indicator indicates the source location of the indicateddirectional sound.

Further aspects of the present disclosure include non-transitorycomputer-readable medium or storage media having embodied thereon aprogram executable by a processor to provide a method for dynamicgeneration and display of play data for the media.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary network environment in which a systemfor detecting gaze to generate virtual effects indicators correlatedwith directional sounds may be implemented.

FIG. 2A illustrates an exemplary uniform data system (UDS) that may beused to provide data to a system for detecting gaze to generate virtualeffects indicators correlated with directional sounds.

FIG. 2B illustrates an exemplary table of various objects and associatedevents, according to an aspect of the present disclosure.

FIG. 3 illustrates an exemplary display with a generated virtual effectindicator.

FIG. 4 illustrates a flowchart of an example method for detecting gazeto generate virtual effects indicators correlated with directionalsounds, according to an aspect of the present disclosure.

FIG. 5 is a block diagram of an exemplary electronic entertainmentsystem that may be used in embodiments of the present invention.

DETAILED DESCRIPTION

Aspects of the present disclosure include systems and methods fordetecting gaze to generate virtual effects indicators correlated withdirectional sounds. Information regarding a plurality of directionalsound types associated with one or more interactive content titles maybe stored in memory. Each directional sound type may be associated withone or more virtual effects indicators in a respective interactivecontent title. Gameplay data sent over a communication network from aclient device of a player engaged in a current activity of one of theinteractive content titles within a current gameplay session may bemonitored. The gameplay data may be indicative of a directional soundassociated with a source location within a three-dimensional virtualenvironment of the interactive content title.

FIG. 1 illustrates a network environment in which a system for detectinggaze to generate virtual effects indicators correlated with directionalsounds may be implemented. The network environment 100 may include oneor more interactive content servers 110 that provide streaming content(e.g., interactive video, podcasts, etc.), one or more platform servers120, one or more user devices 130, and one or more databases 140.

Interactive content servers 110 may maintain, stream, and hostinteractive media available to stream on a user device 130 over acommunication network. Such interactive content servers 110 may beimplemented in the cloud (e.g., one or more cloud servers). Each mediamay include one or more sets of object data that may be available forparticipation with (e.g., viewing or interacting with an activity) by auser. Data about the object shown in the media may be stored by themedia streaming servers 110, platform servers 120 and/or the user device130, in an object file 216 (“object file”), as will be discussed indetail with respect to FIGS. 2A and 3 .

The platform servers 120 may be responsible for communicating with thedifferent interactive content servers 110, databases 140, and userdevices 130. Such platform servers 120 may be implemented on one or morecloud servers. The streaming servers 110 may communicate with multipleplatform servers 120, though the media streaming servers 110 may beimplemented on one or more platform servers 120. The platform servers120 may also carry out instructions, for example, receiving a userrequest from a user to stream streaming media (i.e., games, activities,video, podcasts, User Generated Content (“UGC”), publisher content,etc.). The platform servers 120 may further carry out instructions, forexample, for streaming the streaming media content titles. Suchstreaming media may have at least one object set associated with atleast a portion of the streaming media. Each set of object data may havedata about an object (e.g., activity information, zone information,actor information, mechanic information, game media information, etc.)displayed during at least a portion of the streaming media.

The streaming media and the associated at least one set of object datamay be provided through an application programming interface (API) 160,which allows various types of media streaming servers 110 to communicatewith different platform servers 120 and different user devices 130. API160 may be specific to the particular computer programming language,operating system, protocols, etc., of the media streaming servers 110providing the streaming media content titles, the platform servers 120providing the media and the associated at least one set of object data,and user devices 130 receiving the same. In a network environment 100that includes multiple different types of media streaming servers 110(or platform servers 120 or user devices 130), there may likewise be acorresponding number of APIs 160.

The user device 130 may include a plurality of different types ofcomputing devices. For example, the user device 130 may include anynumber of different gaming consoles, mobile devices, laptops, anddesktops. In another example, the user device 130 may be implemented inthe cloud (e.g., one or more cloud servers). Such user device 130 mayalso be configured to access data from other storage media, such as, butnot limited to memory cards or disk drives as may be appropriate in thecase of downloaded services. Such devices 130 may include standardhardware computing components such as, but not limited to network andmedia interfaces, non-transitory computer-readable storage (memory), andprocessors for executing instructions that may be stored in memory.These user devices 130 may also run using a variety of differentoperating systems (e.g., iOS, Android), applications or computinglanguages (e.g., C++, JavaScript). An example user device 130 isdescribed in detail herein with respect to FIG. 6 .

The databases 140 may be stored on the platform server 120, the mediastreaming servers 110, any of the servers 218 (shown in FIG. 2A), on thesame server, on different servers, on a single server, across differentservers, or on any of the user devices 130. Such databases 140 may storethe streaming media and/or an associated set of object data. Suchstreaming media may depict one or more objects (e.g., activities) that auser can participate in and/or UGC (e.g., screen shots, videos,commentary, mashups, etc.) created by peers, publishers of the mediacontent titles and/or third party publishers. Such UGC may includemetadata by which to search for such UGC. Such UGC may also includeinformation about the media and/or peer. Such peer information may bederived from data gathered during peer interaction with an object of aninteractive content title (e.g., a video game, interactive book, etc.)and may be “bound” to and stored with the UGC. Such binding enhances UGCas the UGC may deep link (e.g., directly launch) to an object, mayprovide for information about an object and/or a peer of the UGC, and/ormay allow a user to interact with the UGC. One or more user profiles mayalso be stored in the databases 140. Each user profile may includeinformation about the user (e.g., user progress in an activity and/ormedia content title, user id, user game characters, etc.) and may beassociated to media.

FIG. 2A illustrates an exemplary universal or uniform data system (UDS)that may be used to provide data to a system for detecting gaze togenerate virtual effects indicators correlated with directional sounds.Based on data provided by UDS, platform server 120 can be made aware ofwhat in-game objects, entities, activities, and events that users haveengaged with, and thus support analysis of and coordination with in-gameactivities. Each user interaction may be associated the metadata for thetype of in-game interaction, location within the in-game environment,and point in time within an in-game timeline, as well as other players,objects, entities, etc., involved. Thus, metadata can be tracked for anyof the variety of user interactions that can occur in during a gamesession, including associated activities, entities, settings, outcomes,actions, effects, locations, and character stats. Such data may furtherbe aggregated, applied to data models, and subject to analytics. Such aUDS data model may be used to assign contextual information to eachportion of information in a unified way across games.

As illustrated in FIG. 2A, an exemplary console 228 (e.g., a user device130) and exemplary servers 218 (e.g., streaming server 220, an activityfeed server 224, an user-generated content (UGC) server 232, and anobject server 226) are shown. In one example, the console 228 may beimplemented on the platform server 120, a cloud server, or on any of theservers 218. In an exemplary example, a content recorder 202 may beimplemented on the platform server 120, a cloud server, or on any of theservers 218. Such content recorder 202 receives and records content(e.g., media) from an interactive content title 230 onto a contentring-buffer 208. Such ring-buffer 208 may store multiple contentsegments (e.g., v1, v2 and v3), start times for each segment (e.g.,V1_START_TS, V2_START_TS, V3_START_TS), and end times for each segment(e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored asa media file 212 (e.g., MP4, WebM, etc.) by the console 228. Such mediafile 212 may be uploaded to the streaming server 220 for storage andsubsequent streaming or use, though the media file 212 may be stored onany server, a cloud server, any console 228, or any user device 130.Such start times and end times for each segment may be stored as acontent time stamp file 214 by the console 228. Such content time stampfile 214 may also include a streaming ID, which matches a streaming IDof the media file 212, thereby associating the content time stamp file214 to the media file 212. Such content time stamp file 214 may beuploaded and stored to the activity feed server 224 and/or the UGCserver 232, though the content time stamp file 214 may be stored on anyserver, a cloud server, any console 228, or any user device 130.

Concurrent to the content recorder 202 receiving and recording contentfrom the interactive content title 230, an object library 204 receivesdata from the interactive content title 230, and an object recorder 206tracks the data to determine when an object beings and ends. The objectlibrary 204 and the object recorder 206 may be implemented on theplatform server 120, a cloud server, or on any of the servers 218. Whenthe object recorder 206 detects an object beginning, the object recorder206 receives object data (e.g., if the object were an activity, userinteraction with the activity, activity ID, activity start times,activity end times, activity results, activity types, etc.) from theobject library 204 and records the activity data onto an objectring-buffer 210 (e.g., ActivityID1, START_TS; ActivityID2, START_TS;ActivityID3, START_TS). Such activity data recorded onto the objectring-buffer 210 may be stored in the object file 216. Such object file216 may also include activity start times, activity end times, anactivity ID, activity results, activity types (e.g., competitive match,quest, task, etc.), user or peer data related to the activity. Forexample, an object file 216 may store data regarding an item used duringthe activity. Such object file 216 may be stored on the object server226, though the object file 216 may be stored on any server, a cloudserver, any console 228, or any user device 130.

Such object data (e.g., the object file 216) may be associated with thecontent data (e.g., the media file 212 and/or the content time stampfile 214). In one example, the UGC server 232 stores and associates thecontent time stamp file 214 with the object file 216 based on a matchbetween the streaming ID of the content time stamp file 214 and acorresponding activity ID of the object file 216. In another example,the object server 226 may store the object file 216 and may receive aquery from the UGC server 232 for an object file 216. Such query may beexecuted by searching for an activity ID of an object file 216 thatmatches a streaming ID of a content time stamp file 214 transmitted withthe query. In yet another example, a query of stored content time stampfiles 214 may be executed by matching a start time and end time of acontent time stamp file 214 with a start time and end time of acorresponding object file 216 transmitted with the query. Such objectfile 216 may also be associated with the matched content time stamp file214 by the UGC server 232, though the association may be performed byany server, a cloud server, any console 228, or any user device 130. Inanother example, an object file 216 and a content time stamp file 214may be associated by the console 228 during creation of each file 216,214.

As shown in the example table 250 of FIG. 2B, such object data (e.g.,the object file 216) may be associated with event information regardingactivity availability change and may be related to other objects withassociated object information. Media-object bindings may form telemetrybetween the objects shown in at least a portion of the live-streamingmedia and the live-streaming media. For example, such object data may bezone data files 252, actor data files 254, mechanics data files 256,game media data files 258, and other gameplay-related data files.

Such object data (e.g., the object file 216) may be categorized as in inprogress, open-ended, or competitive. Such activity data files 216 mayinclude optional properties, such as a longer description of theactivity, an image associated with the activity, if the activity isavailable to players before launching the game, whether completion ofthe activity is required to complete the game, whether the activity canbe played repeatedly in the game, and whether there are nested tasks orassociated child activities. Such activity data files 216 may include anactivity availability change event for, which may indicate a list orarray of currently available activities for the player. For example,this may be used to decide what activities to display in a game plan.

Such zone data files 252 may indicate an area of an associated gameworld with a single coordinate system wherein the zone may have a 2-Dmap associated with it, and may be used to display locations on thezone. If zone data files 252 are applicable, each zone may include azone ID and a short localizable name of the Zone. Such zone data files252 may be associated with a view projection matrix (4×4) to convertfrom 3-D world coordinates to a 2-D map position. Such zone data files252 may be associated with a location change event that indicates anupdate to a current in-game location of the player. Such location changeevent may be posted regularly, or whenever the player's in-game locationchanges significantly. The platform server 120 may store a latest valuein ‘state.’ Such zone data files 252 may include an x, y, z position ofthe player's character in the zone as well as an a, b, c vectorindicating the player's characters orientation or direction. Such zonedata files 252 may be associate with an activity start event and/or anactivity end event and for the activity end event, an outcome ofcompleted, failed, or abandoned may be associated to the activity (e.g.,activity ID).

Such actor data files 254 may be associated with an entity withbehaviors in the game, and can be player-controller or game-controlled,and can change dynamically during gameplay. Such actor data files 254may include an actor ID for the actor, a localizable name for the actor,an image of the actor, and/or a short description of the actor. Suchactor data files 254 may be associated with an actor select event thatindicates that the player's selected actor(s) have changed. The selectedactor(s) may represent the actors the player is controlling in the gameand may be displayed on the player's profile and other spaces via theplatform server 120. There may be more than one actor selected at timeand each game may replace its list of actors upon loading save data.

Such mechanics data files 256 may be associated with an item, skill, oreffect that can be used by the player or the game to impact gameplay(e.g., bow, arrow, stealth attack, fire damage) and may exclude itemsthat do no impact gameplay (e.g., collectibles). Such mechanics datafiles 256 may include a mechanic ID of the mechanic, a short name of themechanic, an image of the mechanic, and/or a short description of themechanic. Such mechanics data files 256 may be associated with amechanic availability change event that indicates that the mechanicsavailable to the player have changed. Available may mean that themechanic is available in the game world for the player to use, but mayrequire the player to go through some steps to acquire it into inventory(e.g., buy from a shop, pick up from the world) before using it. Eachgame may replace its list of mechanics upon loading save data.

Such mechanics data files 256 may be associated with a mechanicinventory change event that indicates that the player's inventory haschanged. Inventory may refer to mechanics that are immediately usable tothe player without having to take additional steps in the game beforeusing it. Inventory information is used to estimate a player's readinessfor various activities, which may be forwarded to the platform server120. Games may replace its list of mechanic inventory upon loading savedata. Mechanics on cool down may be considered part of the inventory.Mechanic counts (e.g., ammunition, healing points) with any non-zerovalue may be treated as “in inventory.” Inventory mechanics may beconsidered a subset of available mechanics.

Such mechanics data files 256 may be associated with a mechanic useevent that indicates that a mechanic has been used by or against theplayer and may be used to be displayed as mechanic usage in a UGCcontext. Such mechanics data files 256 may include a list or array ofmechanics that were used (e.g, fire arrow, fire damage) or whether aninitiator is the player, such that whether the mechanics were used by oragainst the player. Such mechanics data files 256 may include aninitiator actor ID, a current zone ID of the initiator actor, and/or acurrent x, y, z position of the initiator actor. Such mechanics datafiles 256 may be associated with a mechanic impact event that indicatesthat a mechanic had impact on gameplay (e.g., an arrow hit an enemy) andmay be used to display mechanic image in a UGC context. Mechanic use andmechanic image events may be not linked. Such mechanics data files 256may include the initiator action ID, the current zone ID of theinitiator actor, the current x, y, z position of the initiator actor, atarget actor ID, a current zone ID of the target actor, a current x, y,z of the target actor, and a mitigation mechanic that may mitigate theinitiator mechanic.

Such game media data files 258 may be include a game media ID of thegame media, a localizable name for the game media, a media format (e.g.,image, audio, video, text, etc.), a category or type of media(cut-scene, audiolog, poster, developer commentary, etc.), a URL or aserver-provisioned media file, and/or whether the game media isassociated with a particular activity. Such game media data files 258may be associated with a game media start event that indicates that aparticular piece of game media has started in the game right now and agame media end event that indicates that the particular piece of gamemedia has ended.

FIG. 3 illustrates an example display with a generated virtual effectindicator. In a display 300 of a client device, an example third-personperspective of a player 302 in a gaming environment is shown. From theplayer's perspective, there are bats 304 flying on the left and amountain 306 on the right. Hidden behind the mountains is a snake 308,which is hard to see at the moment. However, during gameplay, the soundof the snake 308 can be audibly heard and the directional sound iscoming from the right side. In some examples, the sound is played louderfrom the right speaker or right side of a headset or other headphones.

When a player is hearing impaired, especially if they are hearingimpaired in the right ear, the sound of the snake 308 may not be heard.In such a case, the player may still be looking at the bats 304, forexample, since they are moving around. A camera (not shown) may be usedto detect where the player is looking on the display 300. If it appearsthat during the playing of the sound of the snake 308, the player failsto gaze towards the right, it may be an indication that the player ishearing impaired in the right ear. In such a case, a virtual effectindicator 310 is generated to indicate where the sound is coming from.

As discussed in more detail below, the virtual effect indicator 310 mayonly be generated after a certain threshold of an extent of mismatchbetween a point of focus determined by the gaze data and the sourcelocation. For example, after failing to gaze towards the source locationafter 5 times, then the virtual effect indicator 310 is generated.Furthermore, the player may consistently look a certain degree away fromwhere the source location is. In such a case, the player may have acertain percentage of hearing loss in one or both ears that causes anoffset in hearing. Therefore, a correction angle may be determined basedon gaze patterns and how far off the player is looking from the sourcelocation. Then, the correction angle can be applied to re-balance thesound dynamic between the two speakers or sides of a headset.

FIG. 4 is a flowchart illustrating an example method 400 for detectinggaze to generate virtual effects indicators correlated with directionalsounds. The method 400 of FIG. 4 may be embodied as executableinstructions in a non-transitory computer readable storage mediumincluding but not limited to a CD, DVD, or non-volatile memory such as ahard drive. The instructions of the storage medium may be executed by aprocessor (or processors) to cause various hardware components of acomputing device hosting or otherwise accessing the storage medium toeffectuate the method. The execution of the instructions may beimplemented on the cloud servers (e.g., the steps identified in FIG. 4are performed in the cloud). The steps identified in FIG. 4 (and theorder thereof) are examples and may include various alternatives,equivalents, or derivations thereof including but not limited to theorder of execution of the same.

In step 410, storing information regarding a plurality of directionalsound types associated with one or more interactive content titles inmemory. Each directional sound type is associated with one or morevirtual effects indicators in a respective interactive content title.Directional sound types may vary in direction, kind of sound, and othersonic and location-based parameters. The directional sound types maylater be recorded by the object recorder 206 in the activity file 216. Astatistical analysis may be performed for which directional sound typesare missed by players and can be proactively included in gameplay forknown hearing-impaired players.

In step 420, monitoring gameplay data sent over a communication networkfrom a client device of a player engaged in a current activity of one ofthe interactive content titles within a current gameplay session. Duringgameplay, the collected gameplay data may indicate a directional soundassociated with a source location within a three-dimensional virtualenvironment of the interactive content title. For example, the sourcelocation may be in a display hidden behind another object or too faraway to be seen but can be heard. The directional sound may beexperienced as coming from a side that correlates to where the sourcelocation is on the display using directional audio technology.

In step 430, gaze data is tracked via a camera associated with theclient device during the current gameplay session to identify a point offocus within the three dimensional virtual environment towards which oneor both eyes of the player are focused. The camera may be location on avirtual reality headset, a stand-alone camera set in front of theplayer, or on a console or on the display. The point of focus may bedetermined through various training and calibration modules that adjusta determined point of focus until the player has indicated that thedetermined point of focus is accurate. Alternatively, the calibrationcan be performed by internal measurements determined by a processorbased on known factors.

In step 440, identifying that the point of focus indicated by the gazedata does not move towards the source location within thethree-dimensional virtual environment when the gameplay data indicativeof the directional sound is received. The identifying may includecomparing one or more three dimensional coordinates associated with thesource location of the directional sound with one or morethree-dimensional coordinates associated with the determined point offocus. The identifying may include determining an orientation of theplayer within the three-dimensional virtual environment, and determininga player field of vision that includes one or more three-dimensionalcoordinates of three dimensional virtual environment based on thedetermined orientation, wherein other three-dimensional coordinatesassociated with the three-dimensional virtual environment are excludedfrom the player field of vision. The orientation may be determined basedon a virtual reality headset that detects the orientation of the head orbody of the player. Therefore, in virtual reality, it may be the casethat the sound comes from a location outside of the field of vision andtherefore would require the player to turn. In such a case, the virtualeffect indicator may be an arrow or shading that indicates that theplayer should turn in a specific direction.

Furthermore, the gaze data may be mapped as the point of focus moves toa plurality of locations in the three-dimensional virtual environment.The point of focus indicated by the gaze data may be identified asfailing to move towards the source location within the three-dimensionalvirtual environment as based on the mapped gaze data. Furthermore, aportion of the gaze data sharing a same set of timestamps as theidentified trigger point may be identified as failing to match alocation on a display in which the directional sound is coming from. Athreshold may be set for an extent of mismatch between the point offocus and the source location of the source of the directional sound.The virtual effect indicator may be generated further based on thethreshold being met.

In step 450, generating a virtual effect indicator associated with thedirectional sound type of the indicated directional sound to presentwithin a display of the client device, wherein the virtual effectindicator indicates the source location of the indicated directionalsound. The virtual effect indicator may be displayed dose to where thesource location is or describe where the source location is. Differentvirtual effect indicators may include visual doppler effect centeredaround the source location of the directional sound on the display, awritten cue indicating which direction to look toward the sourcelocation, a visual indicating pointing to the source location of thedirectional sound or a direction of the directional sound, or anotification alerting the player to the source location of thedirectional sound. The virtual effect indicator may be generated as anoverlay or as an object in the gaming environment. The virtual effectindicator may be recorded (or not) by the object recorder 206 and stored(or not) as an activity in the activity file. The virtual effectindicator may be shown or removed from the media file 212 in recordingby the content recorder.

FIG. 5 is a block diagram of an exemplary electronic entertainmentsystem 500. The entertainment system 500 of FIG. 5 includes a mainmemory 505, a central processing unit (CPU) 510, vector unit 515, agraphics processing unit 520, an input/output (I/O) processor 525, anI/O processor memory 530, a controller interface 535, a memory card 540,a Universal Serial Bus (USB) interface 545, and an IEEE interface 550.The entertainment system 500 further includes an operating systemread-only memory (OS ROM) 555, a sound processing unit 560, an opticaldisc control unit 570, and a hard disc drive 565, which are connectedvia a bus 575 to the I/O processor 525.

Entertainment system 500 may be an electronic game console.Alternatively, the entertainment system 500 may be implemented as ageneral-purpose computer, a set-top box, a hand-held game device, atablet computing device, or a mobile computing device or phone.Entertainment systems may contain more or less operating componentsdepending on a particular form factor, purpose, or design.

The CPU 510, the vector unit 515, the graphics processing unit 520, andthe I/O processor 525 of FIG. 5 communicate via a system bus 555.Further, the CPU 510 of FIG. 5 communicates with the main memory 505 viaa dedicated bus 550, while the vector unit 515 and the graphicsprocessing unit 520 may communicate through a dedicated bus 590. The CPU510 of FIG. 5 executes programs stored in the OS ROM 555 and the mainmemory 505. The main memory 505 of FIG. 5 may contain pre-storedprograms and programs transferred through the I/O Processor 525 from aCD-ROM, DVD-ROM, or other optical disc (not shown) using the opticaldisc control unit 570. I/O Processor 525 of FIG. 5 may also allow forthe introduction of content transferred over a wireless or othercommunications network (e.g., 4$, LTE, 3G, and so forth). The I/Oprocessor 525 of FIG. 5 primarily controls data exchanges between thevarious devices of the entertainment system 500 including the CPU 510,the vector unit 515, the graphics processing unit 520, and thecontroller interface 535.

The graphics processing unit 520 of FIG. 5 executes graphicsinstructions received from the CPU 510 and the vector unit 515 toproduce images for display on a display device (not shown). For example,the vector unit 515 of FIG. 5 may transform objects fromthree-dimensional coordinates to two-dimensional coordinates, and sendthe two-dimensional coordinates to the graphics processing unit 520.Furthermore, the sound processing unit 560 executes instructions toproduce sound signals that are outputted to an audio device such asspeakers (not shown). Other devices may be connected to theentertainment system 500 via the USB interface 545, and the IEEE 1394interface 550 such as wireless transceivers, which may also be embeddedin the system 500 or as a part of some other component such as aprocessor.

A user of the entertainment system 500 of FIG. 5 provides instructionsvia the controller interface 535 to the CPU 510. For example, the usermay instruct the CPU 510 to store certain game information on the memorycard 540 or other non-transitory computer-readable storage media orinstruct a character in a game to perform some specified action.

The present invention may be implemented in an application that may beoperable by a variety of end user devices. For example, an end userdevice may be a personal computer, a home entertainment system (e.g.,Sony PlayStation2® or Sony PlayStation3® or Sony PlayStation4®), aportable gaming device (e.g., Sony PSP® or Sony Vita®), or a homeentertainment system of a different albeit inferior manufacturer. Thepresent methodologies described herein are fully intended to be operableon a variety of devices. The present invention may also be implementedwith cross-title neutrality wherein an embodiment of the present systemmay be utilized across a variety of titles from various publishers.

The present invention may be implemented in an application that may beoperable using a variety of devices. Non-transitory computer-readablestorage media refer to any medium or media that participate in providinginstructions to a central processing unit (CPU) for execution. Suchmedia can take many forms, including, but not limited to, non-volatileand volatile media such as optical or magnetic disks and dynamic memory,respectively. Common forms of non-transitory computer-readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, any other magnetic medium, a CD-ROM disk, digital videodisk (DVD), any other optical medium, RAM, PROM, EPROM, a FLASHEPROM,and any other memory chip or cartridge.

Various forms of transmission media may be involved in carrying one ormore sequences of one or more instructions to a CPU for execution. A buscarries the data to system RAM, from which a CPU retrieves and executesthe instructions. The instructions received by system RAM can optionallybe stored on a fixed disk either before or after execution by a CPU.Various forms of storage may likewise be implemented as well as thenecessary network interfaces and network topologies to implement thesame.

The foregoing detailed description of the technology has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or to limit the technology to the precise form disclosed.Many modifications and variations are possible in light of the aboveteaching. The described embodiments were chosen in order to best explainthe principles of the technology, its practical application, and toenable others skilled in the art to utilize the technology in variousembodiments and with various modifications as are suited to theparticular use contemplated. It is intended that the scope of thetechnology be defined by the claim.

What is claimed is:
 1. A method for gaze-based generation of virtualeffects indicators correlated with directional sounds, the methodcomprising: storing information regarding a plurality of directionalsound types associated with one or more interactive content titles inmemory, each directional sound type associated with one or more virtualeffects indicators in a respective interactive content title; monitoringgameplay data sent over a communication network from a client device ofa player engaged in a current activity of one of the interactive contenttitles within a current gameplay session, wherein the gameplay data isindicative of a directional sound associated with a source locationwithin a three dimensional virtual environment of the interactivecontent title; tracking gaze data via a camera associated with theclient device during the current gameplay session to identify a point offocus within the three-dimensional virtual environment towards which oneor both eyes of the player are focused; identifying that the point offocus indicated by the gaze data does not move towards the sourcelocation within the three-dimensional virtual environment when thegameplay data indicative of the directional sound is received; andgenerating a virtual effect indicator associated with the directionalsound type of the indicated directional sound to present within adisplay of the client device, wherein the virtual effect indicatorindicates the source location of the indicated directional sound.
 2. Themethod of claim 1, wherein the virtual effect indicator includes atleast one of a visual doppler effect centered around the source locationof the directional sound on the display, a written cue indicating whichdirection to look toward the source location, a visual indicatingpointing to the source location of the directional sound or a directionof the directional sound, and a notification alerting the player to thesource location of the directional sound.
 3. The method of claim 1,further comprising mapping the gaze data as the point of focus moves toa plurality of locations in the three dimensional virtual environment,wherein identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three-dimensionalvirtual environment is based on the mapped gaze data.
 4. The method ofclaim 1, wherein identifying that the point of focus indicated by thegaze data does not move towards the source location within the threedimensional virtual environment further includes comparing one or morethree dimensional coordinates associated with the source location of thedirectional sound with one or more three-dimensional coordinatesassociated with the determined point of focus.
 5. The method of claim 1,wherein identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three dimensionalvirtual environment further includes: determining an orientation of theplayer within the three-dimensional virtual environment; and determininga player field of vision that includes one or more three-dimensionalcoordinates of three dimensional virtual environment based on thedetermined orientation, wherein other three-dimensional coordinatesassociated with the three dimensional virtual environment are excludedfrom the player field of vision.
 6. The method of claim 1, furthercomprising setting a threshold for an extent of mismatch between thepoint of focus and the source location of the source of the directionalsound, wherein the virtual effect indicator is generated further basedon the threshold being met.
 7. The method of claim 1, further comprisingtracking a plurality of mismatches associated with the player, eachmismatch being between a point of focus and a source location of adirectional sound; and determining a correction angle based on adetermined offset between the points of focus and the source locations,wherein the determined offset indicates hearing loss in one ear wherebythe correction angle corrects for one-sided hearing.
 8. The method ofclaim 1, wherein the identifying further comprises identifying that aportion of the gaze data sharing a same set of timestamps as theidentified trigger point does not match a location on a display in whichthe directional sound is coming from.
 9. A system for gaze-basedgeneration of virtual effects indicators correlated with directionalsounds, comprising: memory configured to store information regarding aplurality of directional sound types associated with one or moreinteractive content titles in memory, each directional sound typeassociated with one or more virtual effects indicators in a respectiveinteractive content title; a communication interface that receivesmonitored gameplay data sent over a communication network from a clientdevice of a player engaged in a current activity of one of theinteractive content titles within a current gameplay session, whereinthe gameplay data is indicative of a directional sound associated with asource location within a three-dimensional virtual environment of theinteractive content title; and a processor that executes theinstructions stored in memory, wherein the processor executes theinstructions to: track gaze data via a camera associated with the clientdevice during the current gameplay session to identify a point of focuswithin the three-dimensional virtual environment towards which one orboth eyes of the player are focused; identify that the point of focusindicated by the gaze data does not move towards the source locationwithin the three-dimensional virtual environment when the gameplay dataindicative of the directional sound is received; and generate a virtualeffect indicator associated with the directional sound type of theindicated directional sound to present within a display of the clientdevice, wherein the virtual effect indicator indicates the sourcelocation of the indicated directional sound.
 10. The system of claim 9,wherein the virtual effect indicator includes at least one of a visualdoppler effect centered around the source location of the directionalsound on the display, a written cue indicating which direction to looktoward the source location, a visual indicating pointing to the sourcelocation of the directional sound or a direction of the directionalsound, or a notification alerting the player to the source location ofthe directional sound.
 11. The system of claim 9, wherein the processorexecutes further instructions to map the gaze data as the point of focusmoves to a plurality of locations in the three dimensional virtualenvironment, wherein identifying that the point of focus indicated bythe gaze data does not move towards the source location within thethree-dimensional virtual environment is based on the mapped gaze data.12. The system of claim 9, wherein identifying that the point of focusindicated by the gaze data does not move towards the source locationwithin the three dimensional virtual environment further includes:comparing one or more three dimensional coordinates associated with thesource location of the directional sound with one or morethree-dimensional coordinates associated with the determined point offocus.
 13. The system of claim 9, wherein identifying that the point offocus indicated by the gaze data does not move towards the sourcelocation within the three dimensional virtual environment furtherincludes: determining an orientation of the player within thethree-dimensional virtual environment; and determining a player field ofvision that includes one or more three-dimensional coordinates of threedimensional virtual environment based on the determined orientation,wherein other three-dimensional coordinates associated with the threedimensional virtual environment are excluded from the player field ofvision.
 14. The system of claim 9, wherein the processor executesfurther instructions to set a threshold for an extent of mismatchbetween the point of focus and the source location of the source of thedirectional sound, wherein the virtual effect indicator is generatedfurther based on the threshold being met.
 15. The system of claim 9,wherein the processor executes further instructions to: track aplurality of mismatches associated with the player, each mismatch beingbetween a point of focus and a source location of a directional sound;and determining a correction angle based on a determined offset betweenthe points of focus and the source locations, wherein the determinedoffset indicates hearing loss in one ear whereby the correction anglecorrects for one-sided hearing.
 16. A non-transitory computer-readablestored medium having instructions embodied thereon, the instructionsexecutable by a computing system to perform a method for gaze-basedgeneration of virtual effects indicators correlated with directionalsounds, the method comprising: storing information regarding a pluralityof directional sound types associated with one or more interactivecontent titles in memory, each directional sound type associated withone or more virtual effects indicators in a respective interactivecontent title; monitoring gameplay data sent over a communicationnetwork from a client device of a player engaged in a current activityof one of the interactive content titles within a current gameplaysession, wherein the gameplay data is indicative of a directional soundassociated with a source location within a three dimensional virtualenvironment of the interactive content title; tracking gaze data via acamera associated with the client device during the current gameplaysession to identify a point of focus within the three-dimensionalvirtual environment towards which one or both eyes of the player arefocused; identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three-dimensionalvirtual environment when the gameplay data indicative of the directionalsound is received; and generating a virtual effect indicator associatedwith the directional sound type of the indicated directional sound topresent within a display of the client device, wherein the virtualeffect indicator indicates the source location of the indicateddirectional sound.
 17. The non-transitory computer-readable medium ofclaim 16, wherein the virtual effect indicator includes at least one ofa visual doppler effect centered around the source location of thedirectional sound on the display, a written cue indicating whichdirection to look toward the source location, a visual indicatingpointing to the source location of the directional sound or a directionof the directional sound, or a notification alerting the player to thesource location of the directional sound.
 18. The non-transitorycomputer-readable medium m of claim 16, further comprising instructionsexecutable to map the gaze data as the point of focus moves to aplurality of locations in the three-dimensional virtual environment,wherein identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three dimensionalvirtual environment is based on the mapped gaze data.
 19. Thenon-transitory computer-readable medium of claim 17, wherein identifyingthat the point of focus indicated by the gaze data does not move towardsthe source location within the three-dimensional virtual environmentfurther includes comparing one or more three-dimensional coordinatesassociated with the source location of the directional sound with one ormore three-dimensional coordinates associated with the determined pointof focus.
 20. The non-transitory computer-readable medium of claim 17,wherein identifying that the point of focus indicated by the gaze datadoes not move towards the source location within the three-dimensionalvirtual environment further includes: determining an orientation of theplayer within the three-dimensional virtual environment; and determininga player field of vision that includes one or more three-dimensionalcoordinates of three dimensional virtual environment based on thedetermined orientation, wherein other three-dimensional coordinatesassociated with the three dimensional virtual environment are excludedfrom the player field of vision.